Document conservative DB Connect / serverless env upgrade rule - #6407

Docs-only follow-up to #6378, and the direction is right — #6378's aggressive bump is exactly the kind of thing worth encoding as a durable rule. No rendered output changes (the comment lives in a {{/* */}} block), so untouched goldens are correct.

That said, I think the new rule file has a few factual problems that would misdirect the next person doing a bump, plus one convention break. Requesting changes on the first three.

1. Rule 3 enshrines an accident as intentional design

Rule 3 says each SQL template "pins its own latest_lts_db_connect_version_spec" and lists it among values that "must NOT be synced." But at this branch's head:

• dbt-sql: >=16.4,<16.5, carrying an inline {{/* TODO: unused — no dbt-sql template references this macro. Remove it. */}}
• default-sql: >=15.4,<15.5

A repo-wide grep finds zero references to latest_lts_db_connect_version_spec outside those two define blocks — it's dead in both templates, and the divergence is a leftover from #6378 bumping one dead copy. The doc reframes that as a deliberate invariant to preserve, which is the opposite of the truth.

The PR description defers dead-macro cleanup in dbt-sql to a separate PR, which is fine — but then this rule shouldn't cite the macro as evidence of intentional divergence. Either drop that clause or say plainly it's dead pending removal.

2. Rules 1 and 2 conflict at the exact decision point the doc exists for

Rule 1: bump DB Connect "only when the pinned DBR release falls out of support — never just to match the newest serverless environment version."

Rule 2: keep the env version, Python pins, and DB Connect "mutually compatible."

When serverless env 6 lands requiring a Python that DB Connect 16.4 doesn't support, Rule 2 forces the bump Rule 1 forbids. That's precisely the next bump this doc is meant to guide, and it gives two contradictory answers. Worth noting #6378's own justification invoked both reasons at once.

Suggest stating the precedence explicitly — e.g. compatibility (Rule 2) is a hard constraint, and Rule 1 governs discretionary bumps within it.

3. Missing .cursor/rules/template-versions.mdc symlink

All 9 existing .agents/rules/*.md files have a matching symlink in .cursor/rules/ (10 .mdc entries = 9 symlinks + 00-agents-context.mdc). This new rule has none, so Cursor users never load it:

ln -s ../../.agents/rules/template-versions.md .cursor/rules/template-versions.mdc

4. The doc hardcodes the numbers it governs

It embeds 5, 16.4, 15.4, and 3.12 in prose, so bumping any pin now requires remembering to edit this doc too, with nothing enforcing it — the rule file goes stale in exactly the scenario it's written for. Prefer describing the invariant and pointing at the .tmpl files as the source of truth. The (`5`) parenthetical in Rule 3 is the clearest case: the shape of the rule ("same value in both") survives a bump, the literal doesn't.

5. Two inaccuracies in the intro paragraph

• It calls the listed five macros "the full set" for default/, but that template also defines latest_databricks_bundles_version.
• "Each bundle template pins the runtime versions ... in library/versions.tmpl" isn't true for default-scala, which pins dbr_version (17.3), databricks_cli_version, scala_version, and java_version in library/template_variables.tmpl. That file isn't matched by the frontmatter globs either, so a version-pin rule silently excludes a template full of version pins. Either widen the globs or scope the opening sentence to the Python/SQL templates.

Smaller notes

• The rewritten .tmpl comment keeps the substantive content (forward-compat, the DBR-15/Python-3.11 rationale) — good, and in line with .agents/rules/comments.md on not dropping context from comments you didn't write. One small regression: the URL loses its "for DB Connect release notes and version compatibility" descriptor and is now a bare trailing link.
• The rule paragraphs run long — 4–6 sentences each after the bolded rule. Existing rules (changelog.md, dresources.md) are tighter, and comments.md's "AI-generated comments trend long and explanatory" applies to rule prose too.
• No .nextchanges/ fragment needed (not user-visible) — correct as-is.

Third round. Verified against 56ed261d5 (rebased on a161e5d56).

Round-1 follow-through first: items 1 (dead latest_lts_db_connect_version_spec cited as intentional divergence — the macro now has zero references repo-wide), 2 (Rules 1↔2 contradiction — compatibility is now explicitly the hard constraint), 3 (.cursor/rules/template-versions.mdc symlink) and 5 (intro inaccuracies) are resolved. Item 4 is partial — see finding 1.

The glob extension is a real fix and I checked it exhaustively: sweeping libs/template/templates/ for hardcoded version literals outside the two library files returns exactly four files, and all four are covered.

Two things I'd like changed before this lands, then a set of one-line edits.

1. The re-attributed 16.4 floor is contradicted by the file it documents

Rule 2 (.agents/rules/template-versions.md:21) now says the pin sits at 16.4 "because that is the lowest DB Connect release whose Python (3.12) satisfies python_version_spec". But libs/template/templates/default/library/versions.tmpl:36 — in this same PR's file — states:

Serverless environment versions 4 and 5, DBR 16, and DBR 17 all use Python 3.12.

If the whole DBR 16 line is Python 3.12, then DB Connect 16.0–16.3 also satisfy >=3.12,<3.13 and are lower than 16.4. (acceptance/localenv/dbconnect-compatible-kept-check uses databricks-connect~=16.0 as a realistic pin, so those releases exist.)

The actual floor is support status: 16.4 is the LTS — latest_lts_dbr_version is 16.4.x-scala2.12 — and 16.0–16.3 are out of support. That is what the comment text this commit deleted said, in a looser form.

This inverts the failure mode the commit set out to fix. Rule 2's headline is "keep the pin as low as compatibility allows" and the body names Python as the binding constraint, so an agent applying it literally can lower the pin to 16.1. Out-of-support appears only as a trigger to raise, never as a floor.

Suggest: the floor is the lowest release that is both still supported (today the 16.4 LTS) and whose Python satisfies the pin — two constraints, and Python alone is not what puts it at 16.4. Dropping the 16.4/3.12 literals while you're there also closes round-1 item 4 and the file's own "the .tmpl files are the source of truth for the current values" promise, which Rule 2 is currently the only paragraph to break.

2. The new glob turns Rule 2 on default-scala, where the policy is the opposite

:5 and :10 add libs/template/templates/**/library/template_variables.tmpl. That file is default-scala's only pin site, and it pins dbr_version 17.3 — the newest LTS — deriving its DB Connect dependency from it:

libs/template/templates/default-scala/template/{{.project_name}}/build.sbt.tmpl:9
libraryDependencies += "com.databricks" %% "databricks-connect" % "{{template `dbr_version` .}}.+"

while default sits at 16.4. So the round-1 glob fix now loads "keep conservative_db_connect_version_spec as low as compatibility allows" onto a template that deliberately does the reverse. Python 3.12 spans 16.x and 17.x, so nothing in Rule 2 stops an agent from reading it as license to downgrade 17.3. Worth scoping Rule 2 to default's macro, or stating the Scala template's policy explicitly.

Relatedly, build.sbt.tmpl:9 is the repo's second DB Connect pin and is open-ended (.+) — the one pin that contradicts Rule 2 outright — and it is named nowhere in the doc. Editing dbr_version does load the rule, so this is a prose gap rather than a coverage gap.

3. ./task test-update-templates does not regenerate everything a pin bump touches

:27. That task runs -run '^TestAccept/bundle/templates' (Taskfile.yml:593), and acceptance subtests are named by directory (acceptance/acceptance_test.go:638, t.Run(dir, …) at :500). Twelve subtests have goldens carrying the pinned values; three are outside that prefix:

• acceptance/pipelines/init/python/output/my_python_project/pyproject.toml:19 — "databricks-connect>=16.4,<16.5", and :5 — requires-python = ">=3.12,<3.13"
• acceptance/pipelines/init/sql/output/my_sql_project/resources/sample_job.job.yml:32 — environment_version: "5"
• acceptance/pipelines/e2e/output/lakeflow_project/pyproject.toml — both pins

These are generated (acceptance/pipelines/init/python/script runs $CLI pipelines init --output-dir output), so a bump changes them and the named command leaves them stale — the agent follows the rule and still gets red CI. .agents/rules/auto-generated-files.md:93 qualifies the same command as "(templates only)" and lists ./task test-update above it for the full sweep; this rule drops the qualifier. Point at ./task test-update, or name both.

This is also the one place the doc restates rather than cites: .agents/rules/testing.md:264 already carries RULE: Use ./task test-update-templates to regenerate materialized templates, and auto-generated-files.md:93 lists it too. Three copies that already disagree on scope is how this finding arose — a cross-reference beats a third restatement.

Smaller

• :19 — Rule 1's "hard constraint" triple omits latest_lts_dbr_version, the DB Connect pin's closest neighbour, and the two have demonstrably drifted: latest_lts_dbr_version reached 16.4 in #3558 while conservative_db_connect_version_spec stayed at 15.4 until #6378. An agent bumping the DBR LTS reads Rule 1 (DBR absent) and Rule 2 ("raise it only when the compatibility constraint above forces it") and concludes nothing forces a DB Connect move.
• :21 — Rule 2 describes the pin as a floor and never explains the <16.5 upper bound. >=16.4 is a conforming edit under "as low as compatibility allows", and rendered into pyproject.toml it resolves 17.x into every freshly initialized project — the opposite of conservative, and the golden diff still starts with 16.4 so it does not read as alarming.
• :23 — Rule 3 omits default_python_version. A future env on Python 3.13 needs python_version_spec, default_python_version (versions.tmpl:47, consumed at src/sample_notebook.ipynb.tmpl:149 as the notebook kernel version) and the hardcoded requires-python; the rule names only environment_version and requires-python, leaving notebook metadata stale. "Grep both" also has no clear antecedent.
• :23 — "several templates hardcode environment_version" is two templates (lakeflow-integrations ×2 files, default-scala ×1). "Two" is more useful than "several" when the instruction is "update every hit".
• :17 — The per-template framing omits the mechanism that makes the sync rule tractable: default-python, default-minimal, lakeflow-pipelines and pydabs are alias dirs — "template_dir": "../default" in their databricks_template_schema.json — so default/library/versions.tmpl is the single pin site for five shipped templates. (default-sql owns a template/ dir but renders no version pins since #6420.) A reader told "most pins are macros in a template's library/versions.tmpl" will grep for a lakeflow-pipelines/library/ that does not exist.
• versions.tmpl:27 — the serverless_environment_version comment links to the DBR runtime release notes (docs.databricks.com/release-notes/runtime/index.html), where no environment version is listed. dbt-sql/library/versions.tmpl:6 and the new rule file both have the right URL. Four lines below the hunk this PR edits, inside the block the PR exists to make legible — worth fixing here.
• :25 — now that dbt-sql matches default at 16.4, Rule 4 has no concrete example left. Fine as-is, just noting the grounding round 1 asked about is gone rather than replaced.
• The comment relocation is defensible — the duplication is what produced the round-1 contradiction. .agents/rules/comments.md:31 is why finding 1 matters: the "why 16.4, not older" context left the comment and arrived misstated.

Verified clean: frontmatter matches the globs + paths convention, symlink matches the other nine, file ends with a newline and has no trailing whitespace, no .nextchanges/ fragment needed, ./task test-update-templates and .agents/rules/auto-generated-files.md both exist, and referenced PRs #3897/#6378 are apt.

One process note, since it has now caught two reviewers of this PR (including me in round 1): claims about dbt-sql's pins go stale fast. At this base, dbt-sql pins 16.4.x-scala2.12 (#6418) and the dead latest_lts_db_connect_version_spec is gone entirely (#6420) — worth re-grepping rather than trusting either review's history.

Findings 1 and 2 are the blocking ones; the rest are one- or two-line edits.